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FOREWORD 

This  paper  was  presented  at  the  20th  Military  Operations  Research 
Symposium,  held  at  the  National  Bureau  of  Standards,  Gaithersburg, 
Maryland,  on  13  December  1967.  It  appears  in  the  proceedings  of  the 
symposium  published  by  the  Military  Operations  Research  Society, 
Alexandria,  Virginia.  The  proceedings  is  a  classified  publication  given 
limited  distribution. 

The  paper  is  reprinted  herein  as  a  Litton  Technical  Paper  to  permit 
its  wider  distribution,  with  permission  by  the  Military  Operations  Research 
Society  to  do  so. 
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HUMAN  FACTORS  IN  WEAPON  SYSTEM  EVALUATION  (U) 

JAMES  K.  ARIMA 

Milliaici  Systias  Oevclipment  Divisun 
Lilian  IMiitnii 


Since  the  beginning  of  the  Military  Operations  Research 
(MORS)  Symposia  there  has  been  considerable  change  In  the 
evaluation  of  weapon  systems.  The  change  has  been  away  from 
stereotyped  testing  under  a  set  of  Ideal,  tightly  controlled  con¬ 
ditions  toward  testing  in  more  operationally  realistic  environ¬ 
ments  which  permit  the  dynamic  interplay  of  the  environment 
and  the  system  being  tested.  This  change  parallels  the  growth 
and  acceptance  of  the  systems  concept  in  the  design  and  de¬ 
velopment  of  weapons.  That  is,  we  no  longer  think  of  a  weapon 
system  In  terms  of  the  shooting  end  of  the  hardware  alone  but 
include  the  people  who  man  and  operate  it.  the  organization  and 
doctrine  for  its  use.  and  the  combat  and  logistics  support 
necessary  for  it  to  be  maximally  effective. 

My  presentation  focuses  attention  on  how  we  have  improved 
the  evaluation  of  the  human  side  of  weapon  systems- -the  area 
commonly  referred  to  as  human  factors.  I  shall  also  mention 
some  shortcomings  as  they  exist  today  and  some  areas  in  which 
considerable  improvement  might  be  expected  before,  say.  the 
30th  MORS  anniversary  symposium.  As  a  footnote,  I  might 
add  that  much  of  what  I  have  to  say  Is  with  reference  to  Army 
systems,  but  I  am  sure  you  will  be  able  to  think  ••»  mparable 
illustrations  in  the  Navy.  Marine  Corps,  or  Air  Force. 


BACKGROUND 

Some  of  the  early  influence  in  getting  human  factors  into 
weapon  system  evaluations  came,  no  doubt,  with  the  testing  of 
a  very  old  system--a  soldier  and  his  rifle.  Information  com¬ 
ing  out  of  World  War  11  and  the  Korean  War  began  to  suggest 
strongly  that  the  way  the  so-called  "average"  soldier  used  his 
rifle  in  combat  did  not  have  much  in  common  with  the  skill 
training  he  had  received.  This  training  resembled  more  the 
type  of  firing  or.e  sees  in  the  annual  rifle  matches  at  Camp 
Perry.  Ohio.  The  biggest  difference,  of  course,  was  that  in 
combat  the  target  could  shoot  bac^  Therefore,  the  firing  of 
his  rifle  meant  that  the  soldier  might  reveal  his  position  and 
draw  the  fire  of  the  enemy.  Moreover,  he  had  only  a  limited 
number  of  rounds  of  precious  ammunition  that  he  felt  compelled 
to  save  for  some  unforeseen  event  that  could  be  extremely  cri¬ 
tical.  And  finally,  the  information  being  gathered  suggested 
that  most  rifle  fire  was  not  aimed — that  is,  there  was  usually 
no  definite,  visible  target,  or.  when  there  was  a  target,  there 
was  often  only  time  to  point  and  fire. 

From  the  training  side,  these  investigations  led  to  the  now 
familiar  Trainfire  concept  with  the  scientific  support  provided 
by  Human  Resources  Research  Office. (HumRRO) .  George 
Washington  University.1  On  the  evaluation  side,  experiments 
such  as  the  SALVO  series  conducted  by  Operations  Research 
Office, (ORO) .  John  Hopkins  University  were  instituted. 

Among  other  things,  these  studies  attempted  to  determine  the 
contingencies  present  when  a  rifleman  fires  his  weapon  in 
combat  and  to  incorporate  the  findings  Into  field  experimenta¬ 
tion.  They  developed  information  relative  to  range,  exposure 


time,  visibility,  movement,  and  signature  or  disclosure  ac¬ 
tivity  of  targets  in  rifle  combat,  and  the  role  of  confusing  con¬ 
texts  in  which  targets  appear.  They  found  that  the  use  of  these 
factors  in  field  expen  mentation  critically  affected  the  accura¬ 
cy  of  rifle  fire,  produced  more  realistic  estimates  of  the  ef¬ 
fectiveness  of  rifle  fire,  and  even  led  to  changing  concepts 
about  the  basic  measures  of  effectiveness  for  rifle  fire. 


Target  and  Range 

The  SALVO- type  ^iproach.  which  emphasizes  targets  and 
target  presentation,  eventually  led  to  the  very  sophisticated 
small  arms  ranges  that  were  developed  at  the  US  Army  Combat 
Developments  Command  Experimentation  Command,  (CDCEC), 
Fort  Ord.  California,  with  the  help  of  its  scientific  contractors, 
the  Stanford  Research  Institute  and  later.  Litton  Systems.  Inc. 


Figure  1  TARGET  IN  COFFIN 


As  shown  in  Figure  1.  the  basic  element  of  the  Urge! 
range  is  a  coffi  -like  box  set  in  the  ground  which  contains  an 
instrumented  pop-up  target  with  sensors  for  hits  and  near- 
misses  and  a  weapon  signature  simulator. 
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Figure  2  SIMULATOR  IN  POSITION 


Figure  2  shown  the  firing  end  of  the  simulator  Just  to  the 
side  of  the  target. 

The  range,  shown  in  Figure  3,  waa  one  of  a  defensive  pos¬ 
ture  with  the  target*  appearing  first  from  the  rear  and  pro¬ 
gressively  "popping-up"  until  at  40  meters  the  closest  target* 
appeared  as  a  unit  to  simulate  an  enemy  assault.  The  defense 
remains  in  a  fixed  position.  The  computer  van  which  houses 
the  computer  and  related  equipment  to  control  target  presenta¬ 
tion.  fire  simulators,  record  hits  and  near  misses,  and  other 
pertinent  data  was  located  Just  off  range.  A  portion  of  the  con¬ 
sole  in  the  van  shows  the  target  actions  and  permits  over-ride 
of  the  computer  in  manually  controlling  the  targets,  if  desired. 


Stressing  the  Individual 

Innovations  in  evaluating  rifle  systems  have  also  empha¬ 
sized  the  person  firing  as  well  as  target  presentation.  I  am 
here  speaking  of  efforts  that  are  usually  referred  to  as  "stress¬ 
ing*'  the  individual.  The  SALVO  experiments,  for  example. 

Inc'  ded  trials  in  which  prone  riflemen  were  instrumented  with 
electrodes  inside  their  boots  to  shock  them.  I  have  been  told 
that  the  shock  was  so  strong  in  some  instances  that  you  could 
see  their  legs  Jumping,  but  there  were  no  great  differences  in 
rifle  fire  measures  between  th*-  shocked  and  nonahockr.i  trials. 

In  another  study,  HumRrlO  used  explosive  charges  planted 
in  the  firing  lane*  which  went  off  in  an  approaching  sequence  so 
that  they  came  closer  and  closer  to  the  individual  firing  down 
the  lane.4  The  implication  was  that  the  charge*  would  eventu¬ 
ally  go  off  right  at  his  firing  position  unless  he  maintained  cer¬ 
tain  target  effects  within  given  time  constraints.  There  were 
slight  differences  in  rifle  fire  accuracy  between  the  charge  and 
no-charge  conditions,  but  the  differences  were  attributed  to  the 
mud.  dirt,  and  debris  thrown  up  by  the  charges,  rather  than  to 
a  stress  effect. 

More  recently,  the  US  Army  Human  Engineering  Labora¬ 
tories.  an  Army  Materiel  Command  (AMC)  agency,  conducted 
a  series  of  studies  on  rifle  fire  In  which  the  stressed  condition 
was  a  frontally  located  BB  gun  firing  at  a  standing  soldier  who 
fired  at  pop-up  targets  beyond  the  BB  gun.4 

There  were  two  no-BB-gun  conditions  in  the  experiment. 

In  one  condition,  the  soldier  fired  in  the  normal  fatigue  uniform 


Figure  3  TARGET  RANGE 


as  showr  in  Figure  4.  All  soldiers  were  instructed  in.  and 
used,  the  pointing  fire  technique. 

Figure  5  shows  a  soldier  dressed  for  firing  in  the  second 
no-BB-gun  condition.  He  wears  the  protective  uniform  that 
was  devised  for  the  experiment.  This  condition  provides  a 
control  for  the  effects  of  the  protective  clothing  alone  on  point¬ 
ing  fire. 

Figure  6  shows  a  soldier  firing  in  protective  uniform  while 
being  shot  back  at  by  the  BB  gun  Note  the  position  of  the  BB 
gun  located  in  the  small  mound  between  the  soldier  and  the  tar¬ 
get  which  has  popped  up  beyond  the  mound.  The  results  showed 
that  there  were  no  major  differences  between  the  first  and  sec¬ 
ond  conditions,  thus  implying  that  the  protective  uniform  did 
not  degrade  firing  performance  under  the  no-BB-gun  condition. 
However,  when  the  BB  gun  was  activated,  performance  change* 
did  occur,  indicating  that  the  BB  gun  caused  the  difference  and 
not  the  clothing. 


Summary 

I  have  used  innovations  in  the  evaluation  of  small  arms  to 
illustrate  ways  in  which  the  contribution  of  human  factors  to 
weapon  system  effectiveness  are  now  more  critically  examined. 
Generally,  we  employ  more  tactically  realistic  situations  or 
settings,  use  personnel  in  the  evaluations  who  normally  man 
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Figure  4  SOLDIER  IN  FATIGUE  UNIFORM 


Figure  6  BB  GUN  CONDITION 


the  system,  and  look  for  ways  to  stress  the  subjects.  You  will 
find  some  attempt  at  incorporating  these  principles  in  weapon 
system  evaluations  being  conducted  over  a  wide  range  of  ac¬ 
tivities,  such  as  those  at  Eglin  Air  Force  Base,  Camp  Lejeune. 
China  Lake,  Point  Mugu,  Sandia  Test  Range,  Holloman  Air 
Force  Base,  and  the  many  test  sites  of  the  AMC  Test  and  Eval¬ 
uation  Command. 


Figure  5  SOLDIER  IN  PROTECTIV  E  UNIFORM 


COMMAND  CONCERN  FOR  HUMAN  FACTORS 

In  some  instances,  there  does  not  seem  to  be  any  pressing 
necessity  for  incorporating  human  factors  into  the  evaluation 
of  weapon  systems.  For  example,  some  systems  are  less  de¬ 
pendent  on  the  human  operator  than  others.  Other  systems  en¬ 
tail  less  stress  on  the  operator  in  the  combat  environment. 

Some  systems,  such  as  those  where  the  human  operator’ s  pri¬ 
mary  function  is  to  monitor  displays,  can  evaluate  human  fac¬ 
tors  in  simulations  that  do  not  exercise  the  entire  system.  And 
then  there  are  systems  that  are  so  expensive  to  operate  that 
genuine  firings  are  devoted  almost  exclusively  to  hardware  as¬ 
pects  and  are  very  limited  in  scope.  Reasons  for  not  consider¬ 
ing  human  factors,  however,  are  more  often  than  not  invoked 
to  sweep  a  difficult  problem  under  the  rug.  But  today,  those 
who  avoid  human  factors  are  more  frequently  being  called  upon 
by  their  superiors  to  account  for  their  actions,  since  the  maxi¬ 
mum,  effective  utilization  of  our  manpower  resources  is  an 
absolute  necessity. 

On  more  than  one  occasion.  General  Abrams,  when  he  was 
Vice  Chief  of  Staff,  queried  CDCEC  as  to  its  approach  and  con¬ 
sideration  of  human  factors.*  On  5  May  1966.  Major  General 
Cagwin.  then  CDCEC  commander,  wrote  the  director  of  his 
research  office:  ’’During  General  Abrams'  visit  the  subject  of 
human  behavior  was  raised  and  discussed  several  times.  The 
discussions  were  quite  revealing  of  the  need  for  a  valid  basis 
of  predicting  human  behavior  under  combat  stress  in  a  broad 
range  of  battlefield  events."  General  Cagwin  also  said  in  his 
letter:  ’’. . .  We  are  concerned  with  human  behavior  as  a  factor 
of  major  importance  to  our  search  for  ’  truth*  by  combat  field 
experimentation.  It  is  highly  desirable- -if  not  imperative — 
that  CDCEC  integrate  human  iactors  into  combat  field  experi¬ 
mentation  as  soon  as  possible." 

In  January  1966,  the  Combat  Developments  Command  (CDC) 
Army  Air  Defense  Agency  solicited  HumRRO' s  help  to  ascer¬ 
tain  the  effects  of  the  combat  environment  on  persons  manning 
Army  air  defense  systems.”’  This  was  after  extensive  Army- 
air  defense  studies  were  criticized  by  the  Harding  Committee 
for  not  sufficiently  considering  the  human  component  of  the 
system . 
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It  ia  apparent  then,  that  while  we  may  have  come  a  long 
way  In  evaluating  human  (actors  in  weapon  system  evaluations, 
there  is  still  considerable  concern  in  the  highest  places  (or  a 
much  greater  and  more  thorough  examination  o { the  role  human 
(actors  play  in  weapon  system  effectiveness, 


CURRENT  SHORTCOMINGS 

What  are  common  shortcomings  today?  On  what  points  can 
weapon  system  evaluations  be  criticized  with  respect  to  human 
(actors?  There  are  several  with  readily  apparent  solutions  that 
I  shall  mention.  The  ones  (or  which  solutions  come  hard  will 
be  left  (or  later. 

The  majority  of  problems  are  associated  with  the  imple¬ 
mentation  of  a  realistic  combat  setting  or  a  challenge  to  the 
individual.  The  most  persistent  shortcoming  lies  in  going  to 
great  lengths  to  create  superficial  realities  of  the  combat  situ¬ 
ation  and  missing  the  elements  that  are  critical  to  the  weapon 
system  being  evaluated. 


Uncertainty  In  Combat  Environment 

One  Important  area  on  the  basis  of  which  evaluations  might 
be  criticized  la  the  amount  of  uncertainty  presented  the  player 
subject.  By  uncertainty,  I  am  apeaklng  In  terms  of  the  entropy 
concept  of  Information  theory.  Translated  to  the  quasi-combat 
acene  of  weapon  system  evaluations,  uncertainty,  then,  refers 
to  the  number  of  probable  or  possible  events  that  confront  the 
Individual  and  the  number  of  possible  courses  of  action  from 
which  the  individual  might  have  to  make  a  choice.  These  two 
aspects  of  uncertainty  In  the  combat  environment  have  been  re¬ 
ferred  to  as  "Information  load"  and  "response  load"  with  re  - 
spect  to  the  individual. 


Information  Load 

fhe  klgher'the  information  load,  the  more  unpredictable 
are  the  events,  and  the  greater  the  uncertainty.  To  Illustrate 
this  point,  let  us  take  as  an  example  the  evaluation  of  a  visually 
sighted,  manually  operated,  antiaircraft  weapon  system  If  the 
target  aircraft  might  appear  from  one  of  two  different  directions 
we  have  one  bit  of  information  load.  But  if  the  aircraft  always 
retraces  a  previous  path  from  the  opposite  direction,  that  is,  It 
comes  one  way  and  goes  back  the  other  way,  we  no  longer  have 
any  uncertainty  and  there  la  no  information  load.  Information 
load.  In  this  hypothetical  setting,  could  be  Increased  in  any 
number  of  ways,  such  as  by  having  several  approach  paths , 
speeds,  altitudes,  cross-over  or  fly-by  patterns,  and  types  of 
aircraft.  Especially  Important  would  be  uncertainty  as  to  when 
the  aircraft  might  appear  and  the  possible  appearance  of  both 
friendly  and  hostile  aircraft. 


Response  load 

By  response  load,  I  refer  to  the  fact  that  individuals  usual¬ 
ly  have  more  than  one  thing  tney  must  do  at  any  one  time. 
Whatever  they  actually  do  represents  a  selection  from  among 
several  alternatives.  When  critical  acta  must  be  taken  In  the 
combat  situation,  the}-  are  often  superimposed  on  the  ongoing 
acts  of  a  fully  occupied  person.  Accordingly,  It  is  usually  de¬ 
sirable  that  individuals  participating  in  a  weapon  system  evalu¬ 
ation  be  engaged  In  sustained,  quasi-combat  operations  as  a 
part  of  the  environment  for  the  teat. 

Another  aspect  of  response  load  lies  In  the  specific  actions 
of  individuals  directly  related  to  operating  the  system  under 
evaluation.  For  example,  multiple  targets  will  present  a  re¬ 
sponse  load  on  the  individual  in  that  he  must  decide  which  tar¬ 
get  to  engage  and  having  decided  to  engage  one,  he  must  attempt 
to  monitor  the  actions  of  the  others  at  the  same  time,  or  a 
display  monitor  may  be  required  to  send  information  while 
monitoring  hla  dlaplays.  Whatever  the  situation,  the  response 


loading  on  individual  participants  should  adequately  reflect  the 
load  present  in  the  tactical  environment.  An  appropriate  re¬ 
sponse  load  on  system  personnel  Is  especially  important  in 
weapon  system  evaluations  where  system  response  character¬ 
istics  such  as  target  selection  and  reaction  times  are  being 
investigated. 

To  summarize,  the  weapon  system  evaluation  situation 
should  attempt  to  remove  any  regularity,  repetition,  and  pre¬ 
dictability  of  events  that  are  not  a  part  of  the  combat  environ¬ 
ment  of  the  system.  It  should  also  provide  adequate  and  ap¬ 
propriate  response  requirements  on  the  personnel  operating 
the  system.  If  the  situation  does  not  provide  adequate  informa¬ 
tion  and  response  loads,  system  effectiveness  may  be  grossly 
overestimated. 


Overcontrol 

Another  problem  area  closely  related  to  those  I  have  been 
discussing  is  the  matter  of  overcontrol  in  weapon  system  evalu¬ 
ations.  This  usually  occurs  because  of  a  misguided  interest 
in  reducing  variability.  The  result  is  a  drastic  reduction  in 
the  information  and  response  loads  for  the  Individual  and  per¬ 
formance  that  Is  grossly  different  from  that  to  be  expected  in 
combat.  The  findings  of  evaluations  conducted  under  such 
tightly  controlled  conditions  are  therefore  very  limited  in  their 
application.  That  is,  the  evaluation  haB  produced  large  amounts 
of  dsta  that  may  not  be  very  pertinent  to  the  many  ways  the 
weapon  Is  actually  used  in  combat  because  of  the  arbitrary  de¬ 
cisions  of  the  evaluator.  There  is,  moreover,  considerable 
danger  that  the  results  may  be  erroneously  taken  to  be  a  valid 
estimate  of  the  effectiveness  of  the  weapon  system  simply  be¬ 
cause  of  the  very  labored  conditions  under  which  the  trials 
were  run.  As  an  example  of  overcontrol,  let  us  take  again  the 
evaluation  of  a  visually  sighted  antiaircraft  system.  Overcon¬ 
trol  would  include  such  things  as  having  the  individual  or  crew 
fre  at  the  approach,  overhead,  and  receding  sectors  of  a  fly¬ 
by  in  separate  blocks  of  trialB.  Overcontrol  would  also  Include 
firing  over  many  successive  trials  at  e  single  range,  a  single 
altitude,  and  a  single  cross-over  pattern  and  the  like.  Sucn 
controls  create  massed  learning  of  a  unitary  set  of  responses 
that  may  eventually  result  in  stereotyped,  unnatural  respond¬ 
ing. 

Overcontrol  also  destroys  the  dynamism  that  is  present  in 
the  tactical  situation  or  system.  For  example,  the  receding 
aircraft,  except  for  unusual  tactical  situations  or  masking  con¬ 
ditions,  represents  a  final  parting  shot  that  the  gunner  has  after 
he  has  failed  to  damage  the  aircraft  on  its  Incoming  and  over¬ 
head  cross-over  path.  This  aspect  is  completely  lost  when  the 
receding  sector  is  fired  as  a  separate  series  of  trials. 


Player  Quality  Control 

Another  area  of  criticism  might  be  the  absence  or  lack  of 
quality  control  of  the  player  personnel.  Too  often,  there  is  a 
tendency  to  devote  a  disproportionate  amount  of  attention  and 
funds  to  physically  measurable  facets  of  a  situation  while  ne¬ 
glecting  such  Important  factors  as  the  representativeness  of  the 
player  sample  from  the  standpoint  of  individual  qualifications 
and  level  of  unit  training.  Appropriate  and  adequate  amounts  of 
training  assume  particular  importance  for  the  evaluation  of  pro¬ 
totype  systems  where  no  training  doctrine  exists.  To  take  for 
example,  an  extreme  case,  let  us  say  that  the  development  of  a 
new  system  stresses  its  accuracy,  as  does  the  evaluation  instru¬ 
mentation.  Let  us  further  say  that  the  old  system  being  replaced 
emphasized  quickness  of  reaction  and  that  the  training  of  troops 
was  heavily  loaded  toward  this  end.  The  grave  error  in  this 
type  of  situation  is  to  borrow  trained  troops  from  somewhere, 
orient  them  briefly  to  the  new  system,  and  then  send  them  out 
for  the  evaluation  of  the  new  system.  Obviously,  the  situation 
does  not  provide  a  good  evaluation  of  the  accuracy  of  the  new 
system. 
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Similarly,  ther;'  le  very  often  a  great  amount  of  attention 
and  money  spent  on  measuring  events  to  an  unrealistic  degree 
of  accuracy  with  complete  disregard,  from  the  human  factors 
standpoint,  of  the  scope  and  purposes  of  the  experiment.  This 
situation  arises  when  the  critical  dependent  measures  arc  based 
almost  wholly  on  the  actions  of  a  human  being,  such  as  target 
detection,  or  when  the  hardware  elements  are  the  same  but 
different  ways  of  utilizing  the  personnel  are  being  evaluated. 

In  such  Instances,  absolute  levels  of  performance  are  not  im¬ 
portant,  and  the  question  asked  is:  "How  much  better  or  worse 
is  one  condition  versus  another?"  When  all  these  conditions 
hold,  the  measure  used  for  comparison  is  usually  a  measure  of 
central  tendency,  and  the  only  concern  about  distributions  of 
these  measures  is  their  relative  homogeneity.  Under  such 
circumstances,  little  change  in  the  measures  of  central  ten¬ 
dency  could  be  expected  regardless  of  whether  a  thousand,  ten 
thousand,  or  a  hundred  thousand  dollars  are  spent  on  the  instru¬ 
mentation.  More  precise  comparisons  can  be  made  in  such 
situations  by  greater  effort  in  the  .  election  and  training  of  the 
player  personnel  to  reduce  variability  on  their  part. 


Instrumentation 

As  long  as  we  are  speaking  of  instrumentation,  I  would 
also  like  to  mention  that  the  instrumentation-man  Interface  is 
too  often  taken  for  granted.  The  instrumentation  and  data  col¬ 
lection  activities  constitute  a  system  that  must  be  evaluated, 
too.  lhe  hardware  might  be  excellent,  but  If  the  hardware 
interfaces  with  the  ongoing  events  of  an  evaluation  through  a 
human  monitor,  the  sensitivity  of  the  hardware  alone  is  not  the 
sensitivity  of  the  data  collection  system.  Some  time  ago,  I 
heard  a  briefing  In  which  it  wan  alleged  that  a  certain  critical 
event  was  being  measured  in  one-hundredths  of  a  second.  This 
was  a  very  misleading  statement  because  the  truth  of  the  matter 
was  that  the  event  being  measured  was  the  thumb  of  a  monitor 
pushing  a  button.  He  eyeballed  the  event  and  had  to  make  a 
decision  us  to  when  the  event  being  measured  was  taking  place. 

In  conjunction  with  another  experiment,  we  actually  did 
measure  the  latency  of  monitor  responses  to  voice  commands.* 
In  this  situation,  the  weapon  operator  gave  five  different  com¬ 
mands  in  sequence,  each  separated  by  several  seconds.  Using 
three  different  buttons  the  monitor  made  three  press  and  two 
release  respons  s.  Our  analysis  showed  no  statistically  sig¬ 
nificant  differed. ces  among  the  responses  so  they  were  pooled. 

In  classic,  laboratory,  reaction-time  -xperiments,  well-prac¬ 
ticed  subjects  who  are  given  consistently  timed  warning  signals 
react  In  the  neighborhood  of  100  to  200  milliseconds  for  key 
pressing  or  key  releasing  responses.  The  pooled  results  of 
2, 700  responses  in  our  study  showed  the  distribution  graphed 
In  Figure  7. 

The  mean  of  the  distribution  is  418  milliseconds.  The 
chape  of  the  distribution  is  characteristic  of  discrimination  re¬ 
action  times  and  seems  to  be  best  fit  by  a  double  monomial  or 
displaced  gamma  function,  according  to  some  recent  research* 

It  is  obvious  that  a  supersonic  aircraft,  for  example,  could 
travel  a  considerable  distance  in  418  milliseconds. 

I  shall  make  one  final  point  In  regards  to  instrumentation. 
This  has  to  do  with  the  use  of  computers,  especially  in  real 
time  experimentation  control  and  data  collection.  Errors  creep 
into  this  type  of  Instrumentation  through  errors  in  communica¬ 
tion  between  the  designer  of  the  evaluation  and  the  computer  pro¬ 
grammer  or  because  the  computer  programmer  is  forced  to 
make  transpositions,  simplifications,  and  other  changes  to  pro¬ 
tocols  as  given  him. 


MAJOR  PROBLEM  AREAS 

1  shall  turn  now  to  those  areas  where  the  problems  of  weap¬ 
on  system  evaluations  loom  large  and  are  more  difficult  to  solve. 
One  of  these  is  the  need  to  get  into  our  evaluations  the  dynamic 
aspects  of  two-sided  engagements,  lids  need  is  especially 
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urgent  where  it  can  be  expected  that  a  weapon  system  will  have 
to  operate  in  direct  confrontation  with  the  enemy. 

By  way  of  example,  let  ua  go  back  to  the  newer  Infantry 
ranges  I  previously  mentioned.  While  they  constitute  a  great 
improvement  in  providing  a  realistic  fire  situation  for  player 
subjects,  It  Is  quite  obvious  that  they  are  still  Just  ranges. 

That  is,  all  the  effects  are  one  way — on  the  target  arrays. 

While  simulators  at  the  targets  fire  to  provide  realistic  weapon 
signatures,  those  firing  the  weapons  do  not  get  killed  or  sup¬ 
pressed  as  the  targets  do. 

Under  such  conditions,  with  a  constant  threat  programmed 
in  the  target  arrays  and  a  common  scenario  or  standard  sched¬ 
ule  of  events.  It  is  possible  to  find  that  a  smaller  infantry  ele¬ 
ment  will  produce  proportionately  greater  target  effects  than  a 
larger  element.  The  smaller  element  may  also  be  easier  to 
control,  It  may  distribute  its  fire  better,  and  it  may  be  more 
efficient  in  ammunition  consumption.  Findings  such  as  these 
arc  constantly  in  danger  of  being  eagerly  accepted  without  due 
consideration  for  the  limitations  present  and  the  many  impor¬ 
tant  unplayed  factors  in  the  evaluation. 

It  is  obvious,  first  of  all,  that  there  is  leas  redundancy  in 
firers  per  target  for  the  smaller  element,  and  accordingly, 
fewer  rounds  are  wasted  In  overkilling  or  ov  r suppressing  tar¬ 
gets.  Also,  a  small  number  of  firers  will  create  leas  flying 
dust  and  debris  in  the  neighborhood  of  targets,  thus  making  for 
better  firing  conditions.  But  the  most  important  factor  not 
played  is  the  expectation  that,  under  these  conditions  of  equal 
threat  and  common  scenario,  the  smaller  unit  would  be  sup¬ 
pressed  and  kill  ;d  more  readily.  For  the  smaller  unit  to  sur¬ 
vive  against  the  uniform  threat,  It  would  have  to  use  different 
and  appropriate  tactics  that  would  significantly  alter  its  fire 
characteristics. 

Simulating  Two-Sided  Combat 

CDCEC  Is  approaching  this  problem  by  evaluating  small 
Infantry  elements  on  the  livefire  range  and  also  on  a  non- 
livefire,  two-sided  combat  experimentation  course.  Fire-effec¬ 
tiveness  parameters  developed  on  the  livefire  range  are  applied 
to  the  blank  fire  used  on  the  combat  experimentation  course  to 
Inflict  casualties.  This  approach  reflects  practical  necessity, 

It  is  a  good  compromise  solution,  and  it  may  be  a  step  that  will 
continue  to  be  necessary  for  some  aspects  of  weapon  system 
evaluations.  But  It  is  obvious  that  problems  Inherent  in  ob¬ 
taining  the  original  livefire  parameters  are  carried  over  into 
the  combat  experimentation  course. 

A  different  way  to  approach  the  two-sided  pr  Mem  in  the 
same  situatin’-,  might  be  to  play  the  livefire  action  sequentially 
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on  the  same  or  two  pieces  ol  very  similar  terrain  for  the  oppos¬ 
ing  forces,  merge  the  one-sided  action,  and  then  repeat  the 
field  situation  with  computer  inflicted  kills  and  suppression  of 
the  firing  elements.  I  shall  refer  to  this  as  the  sequential- 
iterative  method  for  two-sided  combat. 

Figure  8  explains  diagrammatic-ally  how  this  might  be  done. 
In  step  one,  blue  force  using  livefire  fires  on  the  red  force  re¬ 
presented  by  an  appropriate  target  array.  The  computer  stores 
the  results.  The  roles  are  then  reversed.  In  step  two,  red 
force  using  livefire  fires  on  the  blue  force  represented  by  ap¬ 
propriate-targets.  Again,  the  computer  stores  the  results. 

In  step  3.  the  computer,  using  rules  which  have  been  pro¬ 
grammed  into  it,  calculates  a  tentative  schedule  of  casualties 
and  suppression  for  both  sides  in  real  time. 

Figure  9  shows  the  iterative  step  where  the  same  action  is 
replayed.  Only  this  time,  the  computer  Inflicts  casualties, 
causes  suppressions  of  the  firing  elements,  and  controls  the 
simulated  actions  of  targets  to  represent  the  fire  and  movement 
taken  in  steps  one  and  two.  The  computer  stores  the  results 
and  revises  the  casualty  and  suppression  data  to  reflect  what 
happens  in  the  Iteration.  If  the  results  in  steps  4  and  5  are  too 
disparate,  the  situation  Is  reiterated  with  a  revised  casualty  and 
suppression  assessment  program.  If  the  results  are  reasonably 
consistent,  the  action  is  permitted  to  go  on  beyond  the  first 
phase  and  steps  one  through  five  are  repeated  for  the  next 
phase,  and  so  on  until  the  entire  planned  action  is  completed. 
When  the  action  is  completed,  there  is  a  computer  record  of 
the  entire  two-sided  engagement  in  real  time  which  can  be  used 
with  different  weapons,  personnel,  and  doctrinal  parameters  to 
assess  their  effects  in  a  realistic,  empirically  based  simulation. 

Another  possible  way  of  conducting  synthetic  two-sided 
livefire  evaluations  Is  through  a  yoking  technique.  In  concept, 
the  yoking  technique  is  more  simple  than  the  sequential-itera¬ 
tive  technique,  but  there  are  greater  demands  on  the  Instrumen¬ 
tation.  It  la  also  more  desirable  because  it  permits  realtime, 
continuous,  two-sided  action.  Because  of  instrumentation  com¬ 
plexity.  the  technique  would  seem  to  be  best  suited  for  a  duel 
between  two  weapons. 

As  an  example,  let  us  take  a  tank-antitank  weapon  duel. 

Two  identical  pieces  of  terrain  would  be  required.  On  terrain 
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Figure  8  SEQUENTIAL-ITERATIVE  LIVEFIRE  NO.  1 
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Figure  9  SEQUENTIAL-ITERATIVE  LIVEFIRE  NO.  2 


one,  the  tank  woultl  be  live  and  the  antitank  weapon  a  realistic, 
maneuverable,  simulator-firing  target.  On  terrain  two,  the 
antitank  weapon  would  I  e  live  and  the  tank  the  target.  The 
actions  of  the  target  In  each  case  would  be  dictated  through 
telemetry  by  the  actions  of  Its  live  partner,  and  this  represents 
the  concept  of  yoking.  As  the  situation  develop  ,  there  would 
be  two  nearly  identical  duels  going  on  simultaneously. 


Nonrcactlvo  Testing 

The  relatively  fre-  play  that  is  possible  In  any  of  these 
two-sided  methods  is  representative  of  a  concept  termed  "non- 
rcacitve  testing."  which  has  recently  come  to  the  forefront  in 
some  areas  of  behavioral  sciences  research.  It  is  not  unlike 
the  concept  of  nondestructive  testing  in  engineering.  The  basic 
Idea  is  that  the  testing  procedures  should  not  cause  a  reaction 
in  the  process  being  evaluated.  The  key  to  nonreactive  testing 
in  weapon  system  evaluations  is  free  play  of  the  system  being 
evaluated  with  preservation  of  as  much  of  the  two-sided  nature 
of  combat  as  possible  and  Instrumentation  adapted  to  obtain  a 
complete  record  of  events  in  such  a  situation. 


Motivation 

Up  to  IhiB  point,  1  have  assiduously  avoided  the  topic  of 
combat  stress  since  it  was  brought  up  earlier  in  this  presenta¬ 
tion.  I  shall  continue  to  avoid  the  topic.  There  are  too  many 
generalities  and  biases  concerning  the  so-called  "pucker”  factor 
to  treat  the  subject  lightly  and  quickJv ,  It  would  have  to  be  the 
subject  of  a  separate  paper.  Before  dismissing  the  subject,  I 
shall  comment  that  it  does  not  look  as  If  we  Bhall  be  able  to 
duplicate  In  any  experimental  weapon  system  evaluation  the  con¬ 
ditions  which  represent  the  threat  to  life  and  limb  found  in  real 
combat.  Trying  tp  do  so  with  gadgets  and  tricks  have  had  only 
marginal  success.  Moreover,  these  efforts  have  often  clouded 
the  experimental  findings  without  adding  to  them.  This  Is  not 
to  say  that  research  should  not  be  conducted  on  how  to  Induce 
these  effects;  I  am  only  saying  that  the  state  of  the  art  as  it 
presently  exists  is  not  conducive  to  applications, 

On  the  other  hand,  we  can  go  a  long  way  in  making  the 
evaluation  situations  sufficiently  realistic  with  considerable 
stress  placed  on  the  subjects  manning  the  systems.  Loading 
the  individual  with  information  processing  requirements,  pro¬ 
viding  difficult  and  many  courses  of  action  from  which  a  response 
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must  be  selected  and  carried  out,  forcing  the  player  to  reach  a 
decision  rnd  curry  out  an  action  without  suffk  lent  reduction  of 
uncertainty,  and  adding  a  time  demand  on  top  of  all  these  things 
provide  considerable  stress. 

The  element  which  is  missing  from  the  scheme  of  things 
presented  so  far  is  the  steam,  drive,  or  whatever  you  might 
wish  to  call  it  to  make  the  subject  in  an  evaluation  perform 
realistically.  This  is  usually  referred  to  as  a  problem  in  moti¬ 
vation.  However,  even  more  so  than  terms  such  as  fear,  fa¬ 
tigue,  and  stress,  the  concept  of  motivation  has  fallen  into  ill 
repute  because  so  many  undifferentiated  iders  are  included  in 
the  term.  It  is  all  things  to  all  men. 


Incentives  and  Rewards 

Two  aspects  of  motivation  that  can  be  dealt  with  quite 
specifically  arc  incentives  and  rewards,  and  it  is  these  that  I 
would  like  to  finish  with. 

What  is  the  incentive  and  what  is  the  reward  for  a  soldier 
taking  part  in  a  weapon  system  evaluation?  Traditionally,  it  is 
said  that  it  is  his  duty,  once  he  has  been  ordered  to  do  so. 

When  additional  incentives  or  rewards  have  been  proposed,  ser¬ 
ious  objections  have  been  voiced  by  some  unit  commanders. 

They  object  to  giving  the  soldier  something  extra  for  what  he  is 
supposed  to  do.  In  fact,  some  unit  commanders  seem  to  con¬ 
strue  the  proposals  as  a  reflection  on  their  ability  to  get  the 
requisite  performance  out  of  their  troops.  They  may  fail  to 
appreciate  the  fact  that  the  words  ’’duty"  and  "orders"  do  not 
bring  about  results  on  their  own.  They  are  effective  because 
there  is  a  very  generalized  system  of  negative  and  positive 
rewards  that  are  used  throughout  the  services  to  differentiate 
for  the  man  or  woman  in  uniform  those  acts  that  are  appropriate 
in  response  to  the  concepts  of  duty  and  orders.  Negative  re¬ 
wards  include  such  items  as  loss  of  privileges,  extra  duty,  loss 
of  position  in  a  promotion  sequence,  reprimand,  reduction  in 
grade,  loss  of  pay  and  allowances,  confinement,  hard  labor, 
and  many  other  more  subtle  forms  of  punishment.  Positive  re¬ 
wards  include  extra  off-duty  time,  medals,  citations,  promo¬ 
tions,  trophies,  and  the  like.  The  behavior  which  is  molded  or 
shaped  by  these  rewards  is  some  understanding  of  what  is  mini¬ 
mally  necessary  to  avoid  negative  rewards  and  what  is  minimally 
necessary  to  achieve  or  acquire  certain  levels  of  the  positive 
rewards  under  u  host  of  typical  military  situations. 

The  effort  that  individuals,  as  a  group,  will  expend  to  avoid 
negative  rewards  and  acquire  positive  rewards  may  be  used  to 
define  their  incentive  value.  The  effort  that  individuals,  as  in¬ 
dividuals,  will  expend  toward  the  attainment  of  these  rewards 
can  be  used  to  infer  their  prevailing  motivational  orientation. 

At  one  pole,  there  are  those  who  are  motivated  by  a  fear  of 
failure  and  at  the  other  end  there  are  those  who  arc  motivate 
by  a  need  for  achievement.  When  it  comes  down  to  specific  acts 
that  are  desired  of  individuals  at  more  than  a  minimally  accep¬ 
table  level  in  the  ordinary  course  of  duty,  these  very  general 
rewards  1  have  mentioned  have  only  minimal  incentive  value  for 
a  large  portion  of  the  population  that  takes  part  in  weapon  system 
evaluations. 

Accordingly,  over  a  period  of  years,  the  Army,  for  ‘*xam- 
ple,  has  developed  various  forms  of  monetary  incentives  for 
specific  purposes.  There  are  such  things  as  extra  pay  for 
special  skills,  professional  allowances,  superior  pcrfoimance 
pay.  reenlistment  bonuses,  oversea  differentials,  and  hazardous 
duty  pay.  For  a  very  long  time,  the  Army  had  a  suggestion 
awards  program  in  which  civilian  employees  could  get  monetary 
awards,  but  a  soldier  got  a  pat  on  the  back.  It  is  only  relativcl> 
recently  that  the  soldier,  too,  can  receive  monetary  awards. 

So  there  is  a  history  of  using  tangible  incentives  in  the  Army  for 
specific  purposes.  The  usage  has  become  so  commonplace  that 
individuals  sometimes  overlook  the  fact  that  they  exist. 

What  is  the  case  with  military  experiments  and  weapon 
system  evaluations?  Some  evaluations  require  volunteer  sub¬ 


jects;  a  portion  of  a  notice  requesting  volunteers  that  appeared 
in  a  post  daily  bulletin  reads:  "In  addition  to  contributing  to 
our  nation*  s  defense  efforts,  the  program  offers  to  accepted 
volunteers  various  bonuses:  a  four-day  work  week,  a  three- 
day  pass  each  week,  no  fatigue  duty,  and  thirty  dollars  a  month 
per  diem.  "10 

In  Operation  STALK,  which  took  place  in  the  desert  at  Fort 
Irwin,  the  reward  for  the  tank  crew  with  the  best  record  was  a 
3-day,  all  expenses  paid  trip  to  Los  Angeles  with  free  enter¬ 
tainment.  The  second  place  crew  received  an  all-expenses  paid 
trip  to  Los  Angeles  minus  the  entertainment. 

In  other  rare  instances,  cash  has  been  paid  to  subjects,  but 
it  6eems  that  the  3-day  pass  is  the  most  frequently  used  award. 
Escape  from  the  evaluation  situation,  itself,  has  also  been  used, 
either  intentionally  or  unintentionally,  as  a  powerful  incentive. 
That  is,  the  participants  are  informed,  explicitly  or  implicitly, 
that  they  will  be  able  to  leave  the  field  problem  when  a  certain 
level  of  performance  has  been  attaint'd  or  a  certain  amount  of 
work  has  been  completed. 

The  first  comment  I  would  like  to  make  regarding  the  use 
of  incentives  in  this  manner  is  this:  Instead  of  frankly  ack¬ 
nowledging  the  place  and  purpose  of  an  incentive  program  in 
military  experimentation  and  weapon  system  evaluations  and 
developing  a  sound  program,  such  things  as  per  diem  pay,  extra 
time  off,  escape  from  onerous  duty,  "scrounged  up"  rewards, 
and  other  strictly  ad  hoc  measures  are  used  in  a  seemingly  half¬ 
hearted,  almost  surreptitious  manner. 

The  second  point  to  be  made  is  that  these  rewards  result  in 
only  a  first  and  gross  level  of  differentiation  from  the  even  more 
general  rewards  I  have  mentioned  previously  in  respect  to  duty 
and  orders.  That  is,  they  are  used  in  an  indiscriminate  way  to 
boost  or  maintain  an  cverall  level  of  performance.  At  the  same 
time,  this  type  of  reward  accentuates  those  acts  or  performance 
that  are  directly  related  to  attaining  the  reward.  In  civilian 
commerce,  incentive  programs  for  sales  are  often  constructed 
for  such  limited  objectives  as  moving  certain  items  out  of  In¬ 
ventory,  getting  new  clients,  or  increasing  renewal  rates.  The 
monies  that  are  spent  on  such  programs  are  considerable,  but 
they  pay  off.  In  a  military  weapon  system  evaluation,  the  cri¬ 
terion  may  be  the  total  number  of  hits.  One  problem  with  this 
type  of  reward  is  just  this- -it  maximizes  certain  aspects  of  the 
evaluation  to  the  detriment  of  others.  With  total  number  of  hits 
as  a  criterion,  great  amounts  of  ammunition  will  be  used  and 
firing  will  be  indiscriminately  sustained  as  long  as  the  chance 
for  obtaining  a  hit  is  there. 

Another  criticism  that  might  be  made  is  that  some  sort  of 
secondary  system  of  points  is  necessary  in  order  to  score  per¬ 
formance  for  the  overall  prize.  Usually,  the  points  are  only 
given  positively,  and  negative  points  are  not  used.  In  any  case, 
the  player  must  learn  the  rcl?lionship  of  his  performance  to 
points,  l Hit  he  has  no  way  to  determine  how  much  a  point  counts 
towards  the  prize  at  the  time  of  his  performance.  He  can  only 
determine  the  value  of  his  points  by  comparing  them  with  other 
competitors'  points  at  some  time  much  later  when  the  informa¬ 
tion  becomes  available.  If  he  finds  that  he  is  quite  far  behind, 
even  after  having  given  it  a  good  try,  he  will  lose  interest  and 
his  performance  may  deteriorate  more  than  in  a  no  incentive 
situation.  Thus,  while  the  reward  has  succeeded  in  modifying 
behavior  toward  some  evaluation  objectives,  considerable  im¬ 
provement  is  possible  in  the  reward-incentive  system. 

Maximum  improvement  can  be  attained  by  having  reward 
units  that  are  meaningful  to  the  performing  individual  or  ele¬ 
ment  scaled  for  every  encounter  or  aspect  of  performance  that 
is  critical  to  the  evaluation.  These  reward  units  should  provide 
both  positive  and  negative  rewards  (gain  and  loss)  for  greatest 
impact.  The  combat  environment  lor  most  weapon  systems  in¬ 
volves  constant  decision  making  with  possibilities  of  loss  or  gain 
always  a  consequence.  The  incentive  system  should  adequately 
create  these  cognitive  conflicts  for  the  participant. 
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The  most  ready-made,  appropriate  system  of  incentives 
and  rewards  appears  to  be  trading  stamps.  They  have  direct 
and  immediate  value  to  the  participant.  They  occur  in  small 
enough  units  bo  that  cost-gain  considerations  can  be  adequately 
scaled  over  a  wide  range  of  activities.  By  giving  the  partici¬ 
pant  a  credit  in  stamps  at  the  beginning  of  an  evaluation,  real 
loss  can  be  created  by  taking  stamps  away  as  a  penalty  for 
wrong  decisions  and  actions.  Stamps  are  flexible  enough  in 
providing  a  variety  of  prizes  to  satisfy  most  individuals.  They 
do  not  have  to  be  cashed  in  at  any  particular  time.  They  are 
very  individual,  in  that  every  person  can  be  a  winner  or  loser. 
Skillful  promotion  can  increase  the  subjective  value  of  stamps 
to  well  over  their  actual  cash  value.  In  fact,  purchased  in  large 
amounto,  stamp  plans  provide  for  very  cut  rate  merchandise. 

How  can  they  be  used?  Every  round  could  have  a  cost. 
Every  second  that  a  target  remains  undetected  could  have  a 
cost  figure,  perhaps  exponentially  scaled.  Becoming  a  casualty 
under  realistic  rules  of  encounter  could  be  true  opportunity  loss 
with  a  real  cost,  l.e. ,  the  loss  of  opportunity  for  gain  of  stamps 
plus  the  loss  of  the  credit  that  one  started  with.  Positive  re¬ 
wards  of  stamps  can  be  similarly  programmed.  An  overall  vic¬ 
tor  type  of  prize  can  be  incorporated  into  the  schedule  by  putting 
the  penalty  stamps  into  a  pool  for  the  top  performers.  Obvious¬ 
ly,  the  loss  of  a  book  of  stamps  is  a  poor  substitute  for  the  loss 
of  life,  but  the  point  is  (hat  if  the  incentive  loss  were  great 
enough,  the  penalty  severe  and  appropriate  to  acts  leading  te 
the  loss,  the  actions  and  decisions  of  the  participant  would  ap¬ 
proximate  those  made  in  a  combat  situation  more  than  without 
the  Incentive- reward  system. 

I  have  approached  a  large  number  of  incentive  firms  with 
these  ideas,  and  their  replies  have  been  most  enthusiastically 
stated,  not  Just  because  the  incentive  experts  realize  that  there 
is  potential  business  in  the  area,  but  because  they  realize  that 
this  concept  of  carefully  worked  out  Incentive  motivation  is 
quite  feasible,  has  many  potential  areas  of  application,  and 
results  in  the  epitome  of  what  I  might  call  "nonreactive  control." 
That  is,  a  great  deal  of  control  is  possible,  but  the  actions  of 
the  individuals  represent  their  way  of  coping  with  the  problem 
within  the  constraints  expressed.  And  this  Is  precisely  what  is 
desired  for  weapon  system  evaluation  purposes. 

As  a  footnote,  I  must  state  that  the  views  expressed  in  this 
paper  are  entirely  my  own  and  do  not  necessarily  reflect  those 
of  the  Army  or  my  employer,  Litton  Industries. 
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Weapon  systems  have  at  least  a  materiel  component  and  a  human  component.  Evaluation 
of  the  operational  effectiveness  of  weapon  systems  must  fully  involve  both  components. 
The  evaluation  situation  must  be  tactically  realistic,  personnel  who  normally  man  the 
system  should  be  used,  and  there  should  be  an  attempt  to  stress  the  participants.  To 
accomplish  these  goals,  adequate  information  and  response  loads  must  be  provided 
participants,  over-control  should  be  avoided,  and  player  and  instrumentation  personnel 
quality  control  should  be  assured.  Suggestions  are  developed  for  simulating  two-sided 
combat,  the  concept  of  nonreactive  testing  is  introduced,  and  the  role  of  incentives  and 
rewards  to  motivate  and  control  performance  is  discussed. 
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